07. Number of Hidden Units / Layers

"in practice it is often the case that 3-layer neural networks will outperform 2-layer nets, but going even deeper (4,5,6-layer) rarely helps much more. This is in stark contrast to Convolutional Networks, where depth has been found to be an extremely important component for a good recognition system (e.g. on order of 10 learnable layers)." ~ Andrej Karpathy in https://cs231n.github.io/neural-networks-1/

More on Capacity

A more detailed discussion on a model's capacity appears in the Deep Learning book, chapter 5.2 (pages 110-120).